Image processing device that generates an image from pixels with different exposure times

ABSTRACT

An image processing apparatus includes an intermediate image generating unit configured to input an image which has been shot with differing exposure times set by region, generates a plurality of exposure pattern images corresponding to differing exposure times based on the input image, and generates a plurality of timing images which are difference images of the plurality of exposure pattern images; and a distortion correction processing unit configured to generate a corrected image equivalent to an exposure processing image at a predetermined exposure time by synthesizing processing of the plurality of timing images.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation of and claims the benefit under 35U.S.C. §120 of U.S. patent application Ser. No. 13/399,222, titled“IMAGE PROCESSING APPARATUS AND IMAGE PROCESSING METHOD AND PROGRAM,”filed on Feb. 17, 2012, which claims the benefit under 35 U.S.C. §119 ofJapanese Patent Application JP 2011-038240, filed on Feb. 24, 2011. Theentire contents of these applications are hereby incorporated byreference in their entireties.

BACKGROUND

The present disclosure relates to an image processing apparatus andimage processing method and program, and more specifically, it relatesto an image processing apparatus and image processing method and programwherein pixel value reading of a shot image is executed as sequentialreadout processing.

As related art, an overview of two techniques below of exposure controlprocessing will be described in the order of (1) Focal-plane shutteroperation and occurrence of distortion, and (2) Exposure time control(shutter control) to set exposure times by different regions.

(1) Focal-Plane Shutter Operation and Occurrence of Distortion

First, focal-plane shutter operation and the occurrence of distortionwill be described. A shutter operation that controls exposure startingand exposure ending from one direction of an imaging device face is ashutter operation method of image shooting processing with an imagingapparatus. This shutter operation is called a focal-plane shutteroperation or a rolling shutter operation. A feature is that if exposurestarting and exposure ending is controlled from an upper row of theimaging device towards a lower row, for example, exposure time betweenrows shifts slightly.

A CMOS image sensor configuration and shooting processing example willbe described as an example of an imaging device having a focal-planeshutter operation function, with reference to FIG. 1. FIG. 1 is adiagram showing a partial configuration of an imaging device (CMOS imagesensor) 101. The imaging device (CMOS image sensor) 101 is configured tohave a vertical scanning circuit 102, horizontal scanning circuit 103,and multiple pixels 104 that have been disposed in an array.

Within the pixels 104, a charge is accumulated in a photodiode by anexposure processing that accompanies the shooting of a subject. Thecharge accumulated in the photodiode of each pixel is output to avertical signal line 113 via an amplifying transistor and transfertransistor. The signal current output to the vertical signal line 113 isfurther supplied to the horizontal scanning circuit 103, and upon apredetermined signal processing having been executed, is outputexternally via a signal output line 114.

The pixels arrayed vertically are connected in common to the verticalsignal line 113, so in order to independently read out the signals ofeach pixel, only the signal for one pixel should be output at one timeto the vertical signal line 113. That is to say, with the imaging device(CMOS image sensor) 101, as shown in FIG. 2A, for example, a signal isfirst read out from each of the pixels 104 d arrayed in the lowest row,and next, readout is performed from the row of pixels 104 c as shown inFIG. 2B, and subsequently the readout rows are changed and signalreadout is performed, thereby enabling independent readout of signals ofeach pixel. The control signals for the pixel readout are output from ahorizontal reset line 111 and horizontal selection line 112 connected tothe vertical scanning circuit 102 shown in FIG. 1, for example.

The pixels 104 making up the imaging device (CMOS image sensor) eachstart exposure again, immediately following the readout processing ofthe accumulated charge. That is to say, exposure processing for the nextimage frame is started.

Thus, upon the readout processing being executed sequentially by row,and immediately thereafter exposure processing being started,differences in the starting point-of-time and ending point-of-time ofexposure, i.e., a shift in exposure time (or exposure period) occursbetween the photodiode 104 a of the head row and the photodiode 104 d ofthe last row. This is a feature of a shutter operation called afocal-plane shutter operation or rolling shutter operation.

Note that the diagram only shows the four rows of 104 a through 104 d,but this only shows a portion of the imaging device, and an actualimaging device has a great number of rows set, such as several hundredto several thousand rows, and sequential readout is executed by eachrow.

An example of starting and ending timings of exposure of each row andthe charge readout starting timing will be described with reference toFIGS. 3 and 4. FIGS. 3 and 4 both show the temporal axis on thehorizontal axis and the row on the vertical axis.

For example in FIG. 3, the charge readout timing has a timing shiftoccurring by row, as shown with the dotted line 151 a, 151 b shown inthe diagram. Multiple rectangular blocks shown in FIG. 3 show theexposure time of a certain shot image frame, and are the exposure timesby row block made up of a row or multiple rows.

Exposure processing is started immediately following the timing shown inthe readout line 151 a of the first shot image frame. As shown in thereadout line 151 a, the exposure start time becomes time that isslightly shifted for each row. In the graph shown in the diagram, therow on the upper side first has exposure started, and the lower the rowsare, the later the exposure starts. The uppermost row has an exposurestarting time of time (t1), and the bottommost row has an exposurestarting time of time (t2).

The right edges of the multiple rectangular blocks shown in FIG. 3 arethe timings of the readout processing of the exposure image to beexecuted, and accumulated charge of the pixels for the rows is read atthe timings shown by the readout line 151 b.

In this example, the exposure ending time is approximately the readoutprocessing time, and as shown by the readout line 151 b in FIG. 3,readout processing for each pixel is performed by row, sequentially fromthe head row. On the uppermost row, time (t2) is the exposure endingtime, and on the bottommost row, time (t3) is the exposure ending time.Note that with this example, the exposure starting and exposure endingfor each row has the same timing shift for each row, so the exposuretime for all of the rows is the same.

FIG. 4 shows the exposure processing and readout timing corresponding totwo continuously shot frame images at the time of motion shooting. Asshown in FIG. 4, the period of the readout line 152 a through readoutline 152 b is the exposure time for the head shooting frame N, and thepixel value readout is executed from each row at the timing shown byreadout line 152 b.

The period of the readout line 152 b through readout line 152 c is theexposure time for a trailing shot frame N+1, pixel value readout isexecuted for each row at the timing shown by readout line 152 c.

In the example shown in FIG. 4, for the head shot frame N, the exposurestarting time is time (t1a) for the uppermost row and time (t1b) for thebottommost row, and the exposure ending time is time (t1b) for theuppermost row and time (t1c) for the bottommost row. For the trailingshot frame N+1, the exposure starting time is time (t2a) for theuppermost row and time (t2b) for the bottommost row, and the exposureending time is time (t2b) for the uppermost row and time (t2c) for thebottommost row.

In the example shown in FIG. 4, for example the exposure time of thebottommost row of the head shot frame N and the exposure time of theuppermost row of the trailing shot frame N+1 are roughly in the sametimeframe. That is to say, a phenomenon occurs wherein the image data onthe lower side of the head image frame and the image data on the upperside of the trailing frame are images in roughly the same timeframe.

As a result, for example, in the case of imaging a subject havingmovement, or in the case of performing shooting processing such asmoving the camera itself during exposure and shooting, distortion occursin the image from shifts in the exposure time between rows resultingfrom the focal-plane shutter operations.

An example of image distortion will be described with reference to FIGS.5A through 5D. FIG. 5A is a photograph example in the case of shootingwith the camera in a stopped state. FIG. 5B is a photograph example inthe case of shooting while moving the camera in a horizontal direction.The image in FIG. 5A has no distortion occurring, but the image shown inFIG. 5B has distortion occurring.

Similarly, FIG. 5C is a photograph example in the case of shooting whilea car is in a stopped state. FIG. 5D is a photograph example in the caseof shooting while a car is in a moving state. The image in FIG. 5C hasno distortion occurring, but the image shown in FIG. 5D has distortionoccurring.

Such distortion occurs due to shifts in exposure time of the imagingdevices described with reference to FIGS. 3 and 4, i.e., due to exposuretime differing a little at a time from the upper edge row to the loweredge row. The distortion occurrence phenomenon is called a focal-planeshutter phenomenon or a rolling shutter phenomenon.

Related art for reducing such distortion by a focal-plane operation willbe described. For example, Japanese Unexamined Patent ApplicationPublication No. 2004-140479 discloses a method of reducing distortion ofa subject having movement, in which reset operations and readoutoperations by imaging devices are performed at high speed, the imagedata read out at high speed is temporarily stored in a storage device,and the stored data is read out at a slower frame rate and outputdownstream.

The method described in Japanese Unexamined Patent ApplicationPublication No. 2004-140479 has to have high speed readout operationsperforming in order to reduce distortion. However, high speed operationsare restricted, so completely eliminating distortion is impractical.Further, a secondary problem occurs, which is that power consumptionincrease and noise increase occurs due to the high speed operations.

Also, Japanese Unexamined Patent Application Publication No. 2004-140149discloses a technique for adding transistors used for global shutteroperations, inside pixels. However, the disclosed technique in JapaneseUnexamined Patent Application Publication No. 2004-140149 has to have atransistor added so the pixel size of the imaging device increases, andis restricted by not being applicable to use in a small image sensor ora mega-pixel image sensor.

Also, Japanese Unexamined Patent Application Publication No. 2006-148496discloses a configuration to reduce distortion by taking in an outputsignal from an image sensor to a storage apparatus, and generating oneimage from multiple frames.

The method in Japanese Unexamined Patent Application Publication No.2006-148496 will be described using FIG. 6. FIG. 6 shows exposure timeby row for three consecutive shot frames at the time of motion shooting,the three frames being frame N−1, frame N, and frame N+1. As shown inFIG. 6, with an imaging device (CMOS image sensor), shooting is executedwith a focal-plane shutter operation, and exposure timing differs byrow. Accordingly, distortion such as that described earlier withreference to FIGS. 5A through 5D, i.e. image distortion resulting frommovement of the object or the camera itself occurs. Thus, the imagesshot at before and after times are used, interpolation is performedwhich takes time into consideration, and an image is generated andoutput which is equivalent to that wherein an image of all of the rowsof one image frame has been shot at the same time at a certain time.

For example, in the case that the three images of frames N−1 through N+1are shot with the settings in FIG. 6, correction of the image in frame Nis performed, and a corrected image similar to that shot at the sametime as a timing T0, which is in the center position of the shootingtime, is generated. In this event, correction processing is performedwith reference to the image in frame N−1 and the image in frame N+1.

The technique described in Japanese Unexamined Patent ApplicationPublication No. 2006-148496 has the advantage that computation issimple, since an image is generated by linear interpolation betweenframes. However, a storage apparatus (memory) serving as a frame bufferhas to be provided. Also, the processing is not to eliminate distortion,but to cause the distortion to be unnoticeable by blurring thedistortion, so there is a problem in that the screen blurs greatly if anobject or the camera moves greatly.

For example, in this case of an image shot with the settings in FIG. 6,in the generating processing for the corrected image of the frame N, therow on the upper edge of the image is created by interpolation usingweighting of approximately the same amount for each of the image of theframe N and the image of the frame N+1, and the row on the lower edge ofthe image is also created by interpolation using weighting ofapproximately the same amount for each of the image of the frame N andthe image of the frame N+1. By performing such processing, the amount ofblurring due to movement of objects increases at the upper edge andlower edge of the screen. However, the center portion of the screen isapproximately interpolated by the image in frame N, so the center of thescreen only blurs as before, so there is a problem in that the amount ofblurring greatly differs depending on the position on the screen.

Also, Japanese Patent Application No. 2007-208580 discloses aconfiguration to reduce distortion wherein the output signal of theimaging device is temporarily taken into a memory, motion vectors aredetected for each divided region of multiple consecutive photographimages stored in the memory, and one corrected image is generated whileperforming position correction.

With the method in Japanese Patent Application No. 2007-208580, blurringdoes not occur from correction processing as with the above-describedJapanese Unexamined Patent Application Publication No. 2006-148496, butthere are problems in that computation of detecting the motion vectorsis complicated, and in the case that computing the motion vector fails,visually perceivable image breakdown occurs.

Further, Japanese Unexamined Patent Application Publication No.2007-336314 discloses a configuration to reduce distortion withfocal-plane operations, wherein a great number of images takenconsecutively are taken in to the memory by high speed operations of theimaging device, for example, and one corrected image is generated fromthese images.

The method in Japanese Unexamined Patent Application Publication No.2007-336314 is a configuration to generate an image with linearinterpolation, similar to the method in Japanese Unexamined PatentApplication Publication No. 2006-148496, but has the advantage in thatthe amount of blurring in the entire screen due to high speed operationof the imaging device is negligible, and distortion can be correctedwell.

However, with the method in Japanese Unexamined Patent ApplicationPublication No. 2007-336314, high speed operation of the imaging deviceis a premise, so similar to the configuration in the above-describedJapanese Unexamined Patent Application Publication No. 2004-140479,power consumption increase and noise increase become problems.

(2) Exposure Time Control (Shutter Control) to Set Exposure Times byDifferent Regions

Next, exposure time control (shutter control) to set exposure times bydifferent regions will be described. The exposure time as to each pixelof the imaging device can be controlled to expand a dynamic range of theshot image.

In a bright subject region, when the exposure time is long, theaccumulated charge of the pixel saturates, and an accurate pixel valueis not obtained. On the other hand, in a dark subject region, a longerexposure time enables a more accurate pixel value to be obtained,corresponding to the subject brightness. Thus, in a region where thesubject is bright, a pixel value of a pixel set for a short exposuretime is obtained as a valid pixel value. On the other hand, in a regionwhere the subject is dark, the pixel value of the pixel having a longexposure time is obtained as a valid pixel value. These are combined togenerate an output image. Note that at the time of output of a finalpixel value, pixel value adjustment processing based on the exposuretimes is executed.

Japanese Unexamined Patent Application Publication Nos. 2006-253876 and2006-542337, and Japanese Patent Application No. 2008-147818 disclosetechniques to expand the dynamic range of a shot image, setting theexposure times that differ by region of the imaging device. For example,the configuration sets a short time exposure row and a long timeexposure row in every other row of the pixel rows of the imaging device.

For example, Japanese Unexamined Patent Application Publication No.2006-253876 discloses a configuration wherein electronic shutteroperations of a CMOS image sensor are operated with the even-numberedrows and odd-numbered rows operating alternately, thereby settinghigh-sensitivity pixels (long time exposure pixels) and low sensitivitypixels (short time exposure pixels), and enabling imaging of a highdynamic range image by combining pixel values according to the subjectbrightness.

Japanese Patent Application No. 2008-147818 discloses a configurationwherein, in addition to the configuration in Japanese Unexamined PatentApplication Publication No. 2006-253876, modification by row is alsofurther enabled for readout timing.

Japanese Unexamined Patent Application Publication No. 2006-542337discloses a configuration wherein, in an imaging device having a colorfilter with a Bayer array, two patterns of exposure time are set foreach row, or for multiple rows of more than one, and changes are madewith electronic shutter operations.

The configurations in Japanese Unexamined Patent Application PublicationNos. 2006-253876 and 2006-542337, and Japanese Patent Application No.2008-147818 execute exposure time control by region, with aconfiguration using an electronic shutter.

(3) Overview of Related Art

As described above, with a configuration using a focal-plane shutter,shifting in exposure periods by row occurs, for example, and thefundamental problem of distortion occurring that results from thisshifting is not resolved. Also, techniques to perform exposure periodcontrol by region and expand the dynamic range are used, but with theseconfigurations also, the shifting of exposure period by row is notprevented in the case that a focal-plane shutter is used, and theproblem where distortion resulting from this shifting occurs is notresolved.

SUMMARY

As described above, with an imaging device that performs focal-planeoperations, distortion of a shot image occurs when shooting a subjectthat is moving and the like. It has been found desirable to provide animage processing device and image processing method and program thatenables suppression of distortion of a shot image with an imagingapparatus that performs focal-plane operations.

According to an embodiment of the present disclosure, an imageprocessing apparatus includes an intermediate image generating unitconfigured to input an image which has been shot with differing exposuretimes set by region, generate multiple exposure pattern imagescorresponding to differing exposure times based on the input image, andgenerate multiple timing images which are difference images of theplurality of exposure pattern images; and a distortion correctionprocessing unit configured to generate a corrected image equivalent toan exposure processing image at a predetermined exposure time bysynthesizing processing of the plurality of timing images.

The intermediate image generating unit may be configured to input animage having exposure time shifting in increments of regions shot with afocal-plane shutter operation, generate multiple exposure pattern imagescorresponding to different exposure times based on the input image, andgenerate multiple timing images which are difference images of theplurality of exposure pattern images.

The intermediate image generating unit may generate multiple timingimages having exposure times shorter than that of the input image,wherein the distortion correction processing unit synthesizes multipletiming images having exposure times that are shorter than that of theinput image and generates the corrected image.

The distortion correction processing unit may select a timing imagehaving an exposure time that is included in the exposure times of thecorrected image that is to be generated, and executes synthesizingprocessing applying the selected timing image.

For a timing image having all of the exposure times in the exposuretimes of the corrected image to be generated, at the time of computingpixel values of the corrected image, the distortion correctionprocessing unit may perform processing to reflect the pixel values ofthe timing image in all of the pixel values of the corrected image; andfor a timing image having an exposure time overlapping portion with aportion of the exposure times of the corrected image to be generated,the distortion correction processing unit performs processing to reflectthe pixel values of the timing image in the pixel values of thecorrected image according to the ratio of overlapping portions.

The timing image generated by the intermediate image generating unit maybe an image that has been set with exposure times that differ in pixelrow increments; and the distortion correction processing unit selects atiming image having the exposure times included in the exposure times ofthe corrected image to be generated in pixel row increments, andexecutes synthesizing processing applying the selected timing image.

The intermediate image generating unit may generate the timing image ascontinuously shot images with an exposure time shorter than that of theinput image the intermediate image generating unit further including anoutput unit configured to output this timing image generated by theintermediate image generating unit as a high frame rate image.

The intermediate image generating unit may further include multipleintermediate image generating units that execute processing in parallelas to multiple continuously shot images; wherein the distortioncorrecting unit generates a corrected image equivalent to the exposureprocessing image having predetermined exposure times, with thesynthesizing processing of the plurality of timing images generated bythe plurality of intermediate image generating units.

A region, which is a control increment of exposure time of the inputimage, may be one of a pixel block made up of multiple pixels, or a row,or a pixel.

The image processing apparatus may further include a motion detectingunit configured to execute motion detection in region increments of theinput image, wherein the output of the distortion correction processingunit is applied only for the region having motion detected, and anoutput image is generated.

The image processing apparatus may further include an imaging device;and a control unit configured to execute exposure time control in regionincrements of the imaging device.

According to another embodiment of the present disclosure, an imageprocessing method executed by an image processing apparatus, includes:performing, by an intermediate image generating unit, inputting of animage which has been shot with differing exposure times set by region,generating of a plurality of exposure pattern images corresponding todiffering exposure time based on the input image, and generating of aplurality of timing images which are difference images of the multipleexposure pattern images; and generating, by a distortion correctionprocessing unit, a corrected image equivalent to an exposure processingimage at a predetermined exposure time by synthesizing processing of themultiple timing images.

According to another embodiment of the present disclosure, A program toexecute image processing with an image processing apparatus includes:causing an intermediate image generating unit to input an image whichhas been shot with differing exposure times set by region, generate aplurality of exposure pattern images corresponding to differing exposuretime based on the input image, and generate a plurality of timing imageswhich are difference images of the multiple exposure pattern images; andcausing a distortion correction processing unit to generate a correctedimage equivalent to an exposure processing image at a predeterminedexposure time by synthesizing processing of the multiple timing images.

Note that the program according to the present disclosure is a programprovided by a storage medium, for example, as to an informationprocessing apparatus or computer system that can execute various programcodes. Processing according to the program can be realized by executingsuch a program with a program executing unit on an informationprocessing apparatus or computer system.

Further objects, features, and advantages of the present disclosure willbecome clear by the detailed description in the later-describedembodiments and the appended drawings. Note that a system according tothe present Specification is a theoretical collective configuration ofmultiple apparatuses, and is not restricted to apparatuses with variousconfigurations within the same housing.

According to the above-described configurations, an image can beobtained in which distortion, resulting from focal-plane operations orthe like which occurs based on subject movement and so forth, has beensuppressed. Specifically, provided is an image processing apparatusincluding an intermediate image generating unit configured to input animage having exposure time shifting in region increments which has beenshot with a focal-plane shutter operation, for example, generatesmultiple exposure pattern images corresponding to differing exposuretimes based on the input image, and generates multiple timing imageswhich are difference images of the multiple exposure pattern images, anda distortion correction processing unit configured to generate acorrected image equivalent to an exposure processing image at apredetermined exposure time with synthesizing processing of multipletiming images generated by the intermediate image generating unit. Forexample, images without distortion can be generated by selecting onlythe timing images having a predetermined exposure time, and synthesizingthese, in increments of rows for example.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram describing a configuration of an imaging device andimaging processing;

FIGS. 2A and 2B are diagrams describing a configuration of an imagingdevice and imaging processing;

FIG. 3 is a diagram describing an imaging processing sequence by afocal-plane operation;

FIG. 4 is a diagram describing an imaging processing sequence by afocal-plane operation;

FIGS. 5A through 5D are diagrams describing an image distortionoccurrence example as a problem with an imaging processing by afocal-plane operation;

FIG. 6 is a diagram describing a configuration that reduces distortionby generating one image from multiple frames;

FIG. 7 is a diagram describing a configuration example of an imagingapparatus serving as an image processing apparatus;

FIGS. 8A and 8B are diagrams describing a setting example of an exposurepattern;

FIG. 9 is a diagram describing a configuration of a distortioncorrecting unit and a processing example;

FIG. 10 is a diagram describing a configuration of an intermediate imagegenerating unit and a processing example;

FIG. 11 is a diagram describing a specific example of processingexecuted by the intermediate image generating unit;

FIGS. 12A and 12B are diagrams describing a specific example ofprocessing executed by distortion correction processing unit;

FIG. 13 is a diagram describing a specific example of processingexecuted by distortion correction processing unit;

FIGS. 14A through 14C are diagrams describing an example of an exposurecontrol pattern;

FIG. 15 is a diagram describing an example of an exposure controlpattern;

FIG. 16 is a diagram describing an exposure control pattern and asetting example of exposure time and a timing image;

FIG. 17 is a diagram describing a processing example of exposure timecontrol;

FIG. 18 is a diagram describing a configuration example of a distortioncorrecting unit;

FIG. 19 is a diagram describing a distortion correction processingexample that is executed with the configuration of the distortioncorrecting unit shown in FIG. 18;

FIG. 20 is a diagram describing a configuration example of a distortioncorrecting unit;

FIG. 21 is a diagram describing a distortion correction processingexample that is executed with the configuration of the distortioncorrecting unit shown in FIG. 20;

FIG. 22 is a diagram describing output processing of a high speed framerate image that is executed by a distortion correcting unit that has noframe buffer;

FIG. 23 is a diagram describing a configuration example of a distortioncorrecting unit;

FIGS. 24A through 24C are diagrams describing an example of focal-planedistortion occurrence;

FIGS. 25A through 25C are diagrams describing an image exampleapplicable to correction processing of focal-plane distortion accordingto the related art;

FIG. 26 is a diagram describing a correction processing example offocal-plane distortion according to the related art;

FIG. 27 is a diagram describing an image example applicable tocorrection processing of focal-plane distortion according to anembodiment of the present disclosure;

FIG. 28 is a diagram describing a correction processing example offocal-plane distortion according to an embodiment of the presentdisclosure; and

FIG. 29 is a diagram illustrating examples of exposure control patterns.

DETAILED DESCRIPTION OF EMBODIMENTS

Details of the image processing apparatus and image processing methodand program according to the present disclosure will be described belowwith reference to the diagrams. Description will be given in thefollowing order.

1. Configuration Example of Image Processing Apparatus

2. Details of Configuration and Processing of Distortion Correcting Unit

2-1. Overall Configuration and Processing of Distortion Correcting Unit

2-2. Processing of Intermediate Image Generating Unit

2-3. Processing of Distortion Correction Processing Unit

3. Other Embodiments

3-1. Modification of Exposure Control Pattern

3-2. Modification of Exposure Time Control

3-3. Modification of Frame Buffer Setting

3-4. Modification of Distortion Correcting Processing

4. Shared Configuration Example with Other Image Processing

5. Description of Advantages of Processing of the Present Disclosure

1. Configuration Example of Image Processing Apparatus

First, a configuration example of the imaging apparatus serving as anembodiment of an image processing apparatus will be described withreference to FIG. 7. As shown in FIG. 7, the imaging apparatus has anoptical lens 201, imaging device 202 that configured to have a CMOSimage sensor or the like, distortion correcting unit 203 that correctsimage distortion resulting primarily from focal-plane operations, signalprocessing unit 205, and control unit 207.

The incident light enters the imaging device (CMOS image sensor) 203 viathe optical lens 201. The imaging device 202 is an imaging devicewherein sequential readout processing by region, e.g. by row, accordingto a focal-plane operation, is executed.

The imaging device 202 accumulates a charge with the pixels of theimaging device according to the incident light, and outputs aphotoelectric conversion signal based on the accumulated charge to thedistortion correcting unit 203 as image data.

The distortion correcting unit 203 inputs the output image signal fromthe imaging device 202, and corrects the image distortion resulting froma focal-plane operation. This processing will be described in detaillater. The distortion correcting unit 203 outputs the corrected image204 generated by distortion correction to the signal processing unit205.

At the signal processing unit 205, corrected image 204 that has beensubjected to distortion correction is input, and predetermined imageprocessing such as white balance adjusting processing, gamma correctionprocessing and the like, for example, are performed on the input image,and an output image 206 is generated and output.

The imaging apparatus is an imaging apparatus that can photograph movingor still images, and the image device (CMOS image sensor) 202 has asimilar configuration as that described above with reference to FIG. 1.Note that the image device (CMOS image sensor) 202 is an imaging devicethat can control exposure time by region with the control of the controlunit 207.

For example, exposure time control by image region is executed with acertain configuration described in Japanese Unexamined PatentApplication Publication Nos. 2006-253876 and 2006-542337, and JapanesePatent Application No. 2008-147818 described above, or anotherconfiguration according to the related art.

Note that with the examples described below, as an example of theimaging device 202, an example using an imaging device having theconfiguration shown in FIGS. 8A and 8B will be described.

The imaging device 202 has an RGB array as shown in FIG. 8A, and theexposure time (exposure patterns 1 through 4) that differs by units offour pixels is set, as shown in FIGS. 8A and 8B, as

(a) Pixel block with the longest exposure time (Exposure Pattern 1)

(b) Pixel block with the second longest exposure time (Exposure Pattern2)

(c) Pixel block with the third longest exposure time (Exposure Pattern3), and

(d) Pixel block with the shortest exposure time (Exposure Pattern 4).

Pixel blocks with such four patterns of exposure times are set so as tobe adjacent, as shown in FIG. 8A, and are cyclically arrayed. Theexposure time ratio of the exposure patterns 1 through 4 is 4:3:2:1, andas shown in FIG. 8B, the exposure starting time differs by pattern, andreadout times are equal.

2. Details of Configuration and Processing of Distortion Correcting Unit

2-1. Overall Configuration and Processing of Distortion Correcting Unit

Next, the configuration and processing of the distortion correcting unit203 will be described with reference to FIG. 9. As shown in FIG. 9, asensor output image 211 which is an output image from the imaging device202 is input into the distortion correcting unit 203. This sensor outputimage 211 is stored sequentially in the frame buffers 212 and 213. Forexample, when shooting a moving picture, the continuously shot framesare sequentially stored in the frame buffers 212 and 213.

Note that processing can be performed to output one image, using acontinuously shot image at time of still image photography also, and canbe applied to the time of shooting of either still images or movingpictures. If we say that three continuously shot images are frames N−1,N, N+1, and the sensor output image 211 which is the newest shot imageis frame N+1, then the frame images are stored, frame N in the framebuffer 212, and frame N−1 in the frame buffer 213.

The intermediate image generating units 214 through 216 each generatemultiple intermediate images as to the sensor output image 211 and thestored images in the frame buffers 212 and 213. This intermediate imagegenerating processing will be described later.

The multiple intermediate images generated by the intermediate imagegenerating units 214 through 216 are input in the distortion correctionprocessing unit 218. The distortion correction processing unit 218inputs the multiple intermediate images generated by the intermediateimage generating units 214 through 216, and further a scanning lineposition information 217 to be input from the control unit 207 isfurther input, the image distortion from a focal-plane shutter iscorrected, and the corrected output image 219 is generated. Thisprocessing will be described later.

2-2. Processing of Intermediate Image Generating Unit

Next, processing of the intermediate image generating unit configuredwithin the distortion correcting unit 203 shown in FIG. 9 will bedescribed with reference to FIG. 10. The distortion correcting unit 203has three intermediate image generating units 214 through 216, asdescribed with reference to FIG. 9.

The intermediate image generating units 214 through 216 performprocessing as to images that have each been consecutively shot.Specifically, the intermediate image generating unit 214 executesprocessing as to the sensor output image (frame N+1), the intermediateimage generating unit 215 executes processing as to the stored image(frame N) in the frame buffer 212, and the intermediate image generatingunit 216 executes the stored image (frame N−1) in the frame buffer 213.

The processing to execute these three intermediate image generatingunits 214 through 216 differs only in the images to be processed, andbasically are the same processing. Accordingly, the processing of theintermediate image generating unit 214 will be described below as arepresentative example.

A sensor output image 211 is input into the intermediate imagegenerating unit 214. The sensor output image 211 is an image having beenshot, with exposure patterns (exposure patterns 1 through 4) that differby pixel region, i.e., by setting four types of different exposuretimes, as described above with reference to FIGS. 8A and 8B.

First, the interpolating processing unit 222 shown in FIG. 10 inputs thesensor output image 221 that has been set with four exposure times. Thesensor output image 221 shown in FIG. 10 corresponds to the sensoroutput image 211 in FIG. 9. The interpolating processing unit 222performs interpolation processing so that the exposure patterns areobtained for all of the pixels. That is to say, with each of the fourexposure times for all of the pixels of the sensor output image 221,four exposure pattern images 223 through 226, which are equivalent to animage shot with exposure processing having been performed uniformly, aregenerated. Specifically, the four exposure pattern images below aregenerated:

(a) Exposure pattern image 223 that is equivalent to an image shot withexposure pattern 1 having the longest exposure time,

(b) Exposure pattern image 224 that is equivalent to an image shot withexposure pattern 2 having the second longest exposure time,

(c) Exposure pattern image 225 that is equivalent to an image shot withexposure pattern 3 having the third longest exposure time, and

(d) Exposure pattern image 226 that is equivalent to an image shot withexposure pattern 4 having the shortest exposure time; these fourexposure pattern images are generated.

The interpolation processing executed in exposure pattern imagegenerating by the interpolation processing unit 222 is an interpolationmethod by a filter such as linear interpolation or the like, or a methodwherein edge direction detection is performed and interpolation is basedthereupon, or the like. The four exposure pattern images 223 through 226generated by the interpolation processing unit 222 are each input intothree different image generating units 227 through 229, with a pairhaving the least difference in exposure time as one pair.

The processing of the difference image generating units 227 through 229will be described with reference to FIG. 11. The exposure pattern images223 through 226 that are equivalent to the images shot during the fourdifferent exposure times generated by the interpolation processing unit222 shown in FIG. 10 are equivalent to the shot images during the fourdifferent exposure times (4T, 3T, 2T, 1T) as shown in FIG. 11.

The difference image generating units 227 through 229 each input a pairhaving little difference in exposure time from the four exposure patternimages and compute difference pixel values of corresponding pixels, andgenerate the images made up of the difference pixel values thereof astiming images 230 through 232.

The difference image generating unit 227 generates a first timing image230 as below. The first timing image 230 is an image made up of thedifference between a pixel value of the first exposure pattern image 223which is equivalent to an image shot with the exposure pattern 1 whichis the longest exposure time, and a pixel value of the second exposurepattern image 224 which is equivalent to an image shot with the exposurepattern 2 which is the second longest exposure time.

The difference image generating unit 228 generates a second timing image231 as below. The second timing image 231 is an image made up of thedifference between a pixel value of the second exposure pattern image224 which is equivalent to an image shot with the exposure pattern 2which is the second longest exposure time, and a pixel value of thethird exposure pattern image 225 which is equivalent to an image shotwith the exposure pattern 3 which is the third longest exposure time.

The difference image generating unit 229 generates a third timing image232 as below. The third timing image 232 is an image made up of thedifference between a pixel value of the third exposure pattern image 225which is equivalent to an image shot with the exposure pattern 3 whichis the third longest exposure time, and a pixel value of the fourthexposure pattern image 226 which is equivalent to an image shot with theexposure pattern 4 which is the shortest exposure time.

Also, the fourth timing image 233 uses the fourth exposure pattern image226 without change, which is equivalent to the image shot with theexposure pattern 4 which is the shortest exposure time. The fourdifference images correspond to images shot with the settings of theexposure times as shown in FIG. 11. That is to say,

(1) First timing image 230 is an image shot at exposure time T(t0 tot1),

(2) Second timing image 231 is an image shot at exposure time T(t1 tot2),

(3) Third timing image 232 is an image shot at exposure time T (t2 tot3), and

(4) Fourth timing image 233 is an image shot at exposure time T (t3 tot4).

Thus, the four timing images (different images) 230 through 233 areequivalent to the four consecutively shot images at the same exposuretime (T), wherein the shooting timing has shifted by T each time.

Thus, each of the intermediate image generating units 214 through 216shown in FIG. 9 generates four images shot at four different timings asto the input image. That is to say, the intermediate image generatingunit 214 generates four timing images based on the sensor output image(frame N+1).

The intermediate image generating unit 215 generates four timing imagesbased on the stored image (frame N) of the frame buffer 212. Theintermediate image generating unit 216 generates four timing imagesbased on the stored image (frame N−1) of the frame buffer 213.

2-3. Processing of Distortion Correction Processing Unit

Next, processing of the distortion correction processing unit 218 thatis configured within the distortion correcting unit 203 shown in FIG. 9will be described with reference to FIGS. 12A and 12B. The distortioncorrection processing unit 218 inputs a timing image generated by theintermediate image generating units 214 through 216, and correctsdistortion caused by focal-plane shutter operations.

The distortion correction processing here is described with reference toFIGS. 12A and 12B. FIG. 12A is a diagram showing exposure time of threeconsecutively shot images. The vertical axis represents rows and thehorizontal axis represents time. As described above with reference toFIG. 6, in the case of performing shooting processing with a focal-planeoperation, the exposure time shifts sequentially by row of the imagingdevice. FIG. 12A shows a setting example of the exposure time by row ofthe images of frames N−1, N, and N+1, which are three consecutively shotimages.

The images of frames N−1, N, and N+1 which are the three consecutivelyshot images correspond to images that are to be processed with the threeintermediate image generating units 214 through 216 of the distortioncorrecting unit 203 shown in FIG. 9, i.e. correspond to the three imagesto be generated as timing images. That is to say, the intermediate imagegenerating unit 214 generates four timing images based on the sensoroutput image (frame N+1).

The intermediate image generating unit 215 generates four timing imagesbased on the storage image (frame N) of the frame buffer 212. Theintermediate image generating unit 216 generates four timing imagesbased on the storage image (frame N−1) of the frame buffer 213.

FIG. 12B is a diagram showing settings whereby the three intermediateimage generating units 214 through 216 of the distortion correcting unit203 can each distinguish four timing images generated based on theimages of frames N+1 through N−1. As shown in FIG. 12B, the four timingimages of frame N−1 are generated by the intermediate image generatingunit 216 based on the stored image (frame N−1) of the frame buffer 213.

The four timing images of frame N are generated by the intermediateimage generating unit 215 based on the stored image (frame N−1) of theframe buffer 212. The four timing images of frame N+1 are generated bythe intermediate image generating unit 214 based on the sensor outputimage (frame N+1).

In FIG. 12A, three images (frame N−1 through frame N+1) are obtained byrow, and in FIG. 12B, twelve (3×4=12) images (timing images) areobtained by row. The twelve images are equivalent to twelve images shotcontinuously which differ regarding exposure time (T). By generating thetiming images in this way, data can be obtained with finer increments.With this fineness, reduction processing of distortion from afocal-plane shutter can be performed with higher precision that could beperformed with the related art.

An example of processing executed with the distortion correctionprocessing unit 218 will be described with reference to FIG. 13. FIG. 13is a diagram that is similar to the diagram shown in FIG. 12B. Thediagram shows settings whereby the three intermediate image generatingunits 214 through 216 of the distortion correcting unit 203 can eachdistinguish four timing images generated based on the images of framesN+1 through N−1.

For example, processing to generating an image with the output timing(Tx) shown in FIG. 13, i.e. a processing example in the case ofgenerating an image shot wherein the exposure time is t1 through t2=Tx,will be described.

Processing to compute the pixel values of the pixels on the row of theupper edge making up the image at the output timing (Tx) shown in thisFIG. 13 will be described. In the event of image generating processing,timing images A, B, C, D, E shown in FIG. 13 will be used. Of theshooting timing of timing images B, C, D, all periods are included in arange of output timing (Tx) of the image planned to be generated.Accordingly, these timing images B, C, D are used without change.

On the other hand, the shooting timing of timing images A and E ispartially overlapping with the output timing (Tx) of the image plannedto be generated and partially not overlapping. Accordingly, blendingprocessing is performed which multiplies these by a predeterminedweighting coefficient.

The pixel values of the pixels on the row of the upper edge (OUT) makingup the image at the output timing (Tx) shown in FIG. 13 are computedwith the following (Expression 1)OUT=a×A+B+C+D+(1−a)×E  (Expression 1)

where A through E are pixel values at the same pixel position as thevarious timing images, i.e. at the corresponding pixel position, and ais a weighting coefficient.

Note that the weighting coefficient a sets the value that is equivalentto the overlapping ratio of the output timing (Tx) shown in FIG. 13 andthe various timing images, for example. Also note that the weightedcoefficient a changes depending on the exposure timing of each row, sohas to have a scanning line position information 217.

In the above (Expression 1), the timing images B, C, D use the pixelvalues 100% without blending and without change, and so the blurring byinterpolation processing can be minimized.

As shown in FIG. 13, the timing images generated by the intermediateimage generating unit 214 through 216 are images wherein the exposuretime differing in increments of pixel rows is set, and the distortioncorrection processing unit 218 selects the timing image having theexposure time that is included in the exposure time of the correctedimage planned to be generated in increments of pixel rows, and executessynthesizing processing to which the selected timing image is applied.

Pixel value computing processing for rows on the upper edge has beendescribed above, but for example in the case of computing pixel valuesof an image at output timing (Tx) shown in FIG. 13 for rows on the loweredge, processing using timing images P, Q, R, S that are shown in FIG.13 is performed.

The pixel value (OUT) of the pixel on the lower edge row can be foundwith (Expression 2) belowOUT=P+Q+R+S  (Expression 2)

where P through S are pixel values at the same pixel position as thevarious timing images, i.e. the pixel values at corresponding pixelimage positions.

In this case, the total of the exposure timings of the four timingimages exactly matches the output timing (Tx), and the timing imageoverlapping with the output timing (Tx) does not have to be used. Otherrows can also similarly have the pixel values of the output imagecomputed with pixel value adding processing of multiple timing imagesusing the pixel value (OUT) computing expression.

Thus, the distortion correction processing unit 218 synthesizes multipletiming images to compute the pixel values of the corrected images, andgenerates and outputs the corrected image 204.

As we can see from the processing described with reference to FIG. 13herein, the distortion correction processing unit 218 generates thecorrected image 204 with the synthesizing processing of multiple images(timing images). The exposure time of the timing image is an imagehaving ¼ the exposure time of the original shot image, and the correctedimage 204 is generated with synthesizing processing that combines these.That is to say, from the upper edge row to the lower edge row, acorrected image 204 using a timing image that is included in roughly thesame exposure time (Tx) can be generated.

Consequently, the output image can be generated as an image shot withsettings of roughly the same exposure time (Tx) for all of the rows fromthe upper edge row to the lower edge row.

Accordingly, the output image generated with the present method has verylittle occurrence of image distortion resulting from focal-planeoperations such as described above with reference to FIGS. 5A through5D. The image distortion resulting from focal-plane operations resultsfrom shifts in exposure time between the upper edge row and lower edgerow, but by performing synthesizing processing of the timing image shownin FIG. 13, an image wherein all of the rows from the upper edge row tothe lower edge row are shot with the settings of approximately the sameexposure time (Tx) can be generated.

Consequently, the corrected image 204 generated by the distortioncorrecting unit 204 becomes an image that has suppressed imagedistortion.

As shown in FIG. 7, the corrected image 204 output by the distortioncorrecting unit 203 subsequently is output to the signal processing unit205 and subjected to predetermined signal processing such as whitebalance adjustments and γ correction and so forth, then output as anoutput image 206.

3. Other Embodiments

Next, other embodiments will be described.

3-1. Modification of Exposure Control Pattern

In the above-described examples, an example is described wherein settingis performed for four exposure times in increments of pixel blocks of arectangular region, with reference to FIGS. 8A and 8B as exposurecontrol patterns of an imaging device.

The setting patterns for exposure regions and exposure times are notrestricted to such settings, and other various types of settings can bemade. Regions to be control increments of exposure time of the inputimage may be set variously, such as pixel blocks made up of multiplepixels, or rows, or pixels.

Note that in the above-described example, the number of exposure controlpatterns is four, but the present disclosure can be implemented if thereare two or more patterns. An example of changing two exposure patternsis shown in FIGS. 14A through 14C. FIG. 14A is an example of changingthe electronic shutter operation every two rows and alternately settingtwo exposure times in increments of two rows. FIG. 14B is an example ofchanging the electronic shutter operation every two by two pixels andsetting the two exposure times every two by two pixels. FIG. 14C is amethod of driving by changing the electronic shutter operation infurther different increments. The example shows settings whereby threepixels in the vertical direction and three pixels in the horizontaldirection are alternately selected and a region is set, and the twoexposure times of the regions are alternately set. Setting variousexposure patterns in this manner can be performed.

Also, the pixel array described with reference to FIGS. 8A and 8B aboveand the pixel array shown in FIGS. 14A through 14C both have a pixelcolor filter in a Bayer array, but a pixel array that is for example anon-Bayer array of a pixel array as shown in FIG. 15 can also beapplied.

The pixel array shown in FIG. 15 is a configuration wherein the pixelarray has W (white) pixels in addition to RGB. Note that in the casethere are two exposure patterns, for example as shown in FIG. 16, theexposure time ratio of the exposure time of exposure pattern 1 andexposure time of the exposure pattern 2 can be set as 2:1. The followingholds true for the distortion correcting unit 203, as shown in FIG. 16.

(1) First timing image=exposure pattern 1−exposure pattern 2

(2) Second timing image=exposure pattern 2

These two timing images can be generated, and a corrected image thatreduces the distortion by synthesizing processing of a timing imagesimilar to the image generating processing described with reference toFIG. 13 above can be generated.

3-2. Modification of Exposure Time Control

With the above-described example, a setting example wherein exposurestarting is shifted in region increments, and the readout timings arecoordinated, has been described as shooting processing in the case ofobtaining imaging images having different exposure times, but theexposure time control by region increments can have various settings.

For example, as shown in FIG. 17, the readout timing can be shifted toobtain the image. Additionally, with the exposure time control in regionincrements as described in Japanese Unexamined Patent ApplicationPublication Nos. 2006-253876 and 2006-542337, and Japanese PatentApplication No. 2008-147818 described above, or with other controls, animage can be shot with settings to periodically perform control ofexposure time or readout timing, by row or by pixel, and processing ofimages shot with such various controls can be performed.

3-3. Modification of Frame Buffer Setting

With the configuration described above with reference to FIG. 9, anexample using two frame buffers is shown, but a configuration may useone frame buffer as shown in FIG. 18, and may use only two consecutivelyshot images. With this configuration, storage apparatus capacity andreading/writing bands can be reduced.

Note that in the case of using this configuration, the distortioncorrection processing unit 218 generates a distortion corrected imageusing a timing image generated by applying only two consecutive framesas shown in FIG. 19, and not the processing applying the three frames offrame N−1 through frame N+1 as shown in FIG. 13.

Further, the present disclosure may have a configuration wherein a framebuffer is not used, as shown in FIG. 20. Note that in the case of usingthis configuration, the distortion correction processing unit 218generates a distortion corrected image using multiple timing imagesgenerated by applying only one frame as shown in FIG. 21, and not theprocessing applying the three frames of frame N−1 through frame N+1 asshown in FIG. 13.

In this case, the setting of the exposure time period of an image outputas shown in FIG. 21 results in a shorter time than the frame rate of theinput image, whereby noise effects can occur. Also, as shown in FIG. 21,for example a timing image which includes exposure period of the data ofthe latter half portion of the output image timing for the upper edgerow does not exist. Similarly, a timing image which includes exposureperiod of the data of the earlier half portion of the output timing forthe lower edge row does not exist. Consequently, an image is generatedjust from a nearby timing image, whereby distortion reduction advantagesby the focal-plane shutter operations can be thinned down slightly onthe upper and lower edges. However, there is an advantage in thatdistortion can be corrected without using a frame buffer.

Further, the distortion correcting unit 203 can also generate an signalof a frame rate that is higher speed than the frame rate of the inputimage. That is to say, the intermediate image generating unit generatesa timing image as a consecutively shot image having an exposure timethat is shorter than the input image, and makes up an output unitwhereby the timing image generated by the intermediate image generatingunit is output as a high frame rate image, thereby enabling a frame ratesignal that is higher speed than the frame rate of the input image to begenerated and output.

Specifically, for example, as shown in FIG. 22, by generating an imagewith optional timing while performing correcting that eliminatesfocal-plane distortion as to the input image (frame N, N+1, . . . ), animage can be output that has a high frame rate of the output image (M,M+1, M+2, M+3, . . . ). Note that in the case of performing imageoutput, performing predetermined image correcting such as brightnesscorrecting and so forth and outputting is preferable.

3-4. Modification of Distortion Correction Processing

With the examples described above, an example using FIG. 13 and(Expression 1) have been described as correction processing performedwith the distortion correcting unit 203, but applying a correctionmethod using a method other than such a linear interpolation may beused.

For example, motion is detected from an image A and an image B which aretwo adjacent timing images shown in FIG. 13, and with motioncompensation as to the image A, a motion compensation timing image A′that is premised on being shot at the same timing as image B isgenerated.

Similarly, motion is detected from an image D and an image E, and amotion compensation timing image E′ is generated.

These images are used to compute the pixel value (OUT) of the correctedimage, according to (Expression 2) shown below.OUT=a×A′+B+C+D+(1−a)×E′  (2)

where A′, B, C, D, E′ are pixel values at the same pixel positions ofthe various timing images or motion compensation timing image, i.e. ofcorresponding pixel positions, and a is a weighting coefficient.

Note that the weighting coefficient a sets a value that is equivalent toa overlapping rate between the output timing (Tx) shown in FIG. 13 andthe various timing images, for example. Also note that the weightingcoefficient a changes depending on the exposure timing of the rows, sothere has to be scanning line position information 217.

4. Shared Configuration Example with Other Image Processing

The present disclosure includes a method to reduce distortion resultingfrom focal-plane shutter operations, but by using other processing alsoat the same time, image quality can be further improved. Several of suchconfiguration examples will be described.

In the example described above, in order to correct focal-planedistortion as shown in FIG. 10, processing is performed with theintermediate image generating unit to interpolate each pixel array ofdifferent exposure controls, and images are generated of the exposurepatterns (first exposure pattern image 223 through fourth exposurepattern image 226) at all pixel positions.

Such processing does not cause a problem for images that are blurred dueto motion that causes distortion, but a problem occurs in that, whenshooting a subject that is completely still, the resolutiondeteriorates.

A configuration example of the distortion correcting unit 203 to reducefocal-plane distortion while preventing such resolution deteriorationwill be described with reference to FIG. 23.

The sensor output image 211 is an image with exposure times that differby pixel. A gain compensation processing unit 241 performs processing tomultiply the gain according to exposure time in region increments of thesensor output image 211 by the pixel values.

For a still image, an image having no resolution deterioration that isthe same as the normal Bayer array can be obtained by executing the gaincompensation processing. The output of the gain compensation processingunit 241 has distortion, and blurring amounts of the motion for eachexposure control pattern differs, whereby in the case there is motion,image breakdown occurs.

Therefore, motion is detected by pixel or by area with the motiondetecting unit 242, and with a motion adapting processing unit 243,selecting processing is performed such that an output image of thedistortion correction processing unit 218 is used in a pixel regionhaving motion, and an output image of the gain compensation processingunit 241 is used in a location having no motion, or blending processingis performed according to the motion amount. Thus, the resolution of animage having no motion is as it has been in the past, and locationshaving motion can reduce the distortion.

Further, portions having no motion use a frame memory to perform noisereduction processing in the temporal direction (also calledthree-dimensional NR or 3DNR), thereby reducing noise in the still imageportions. Further, a configuration may be made which adds a pixel valuesaturation countermeasure processing.

In FIG. 11 described above, in order to generate the timing images 230through 233, the difference in exposure pattern images that differ inthe difference image generating units 227 through 229 is calculated, butfor example in the case that the pixel value of one of the images to besubjected to difference calculation is saturated, an accurate differencevalue is not be obtained.

For example, in the case that the corresponding pixel values of the twoimages applied to generate the timing image (difference image) areoriginally 1200 and 800,

Difference image pixel value=1200−800=400 holds true. However, in thecase that the output of the sensor is 10 bit, the sensor output can onlyoutput the pixel values (0 through 1023). In this case the pixel value2000 mentioned above is output as pixel value 1023,

Difference image pixel value=1023−800=223 holds true, and a timing image(difference image) having a pixel value smaller than the actual can begenerated.

As a saturation countermeasure, the simplest is to perform clippingprocessing with values that differ according to exposure patterns.

As shown in FIG. 11, the exposure ratio of the exposure patterns 1, 2,3, 4 is 4:3:2:1, and in the case the pixel value output of the imagingdevice is 10 bit, after clipping the pixel in exposure pattern 1 with1023, the pixel in exposure pattern 2 with 768, the pixel in exposurepattern 3 with 512, and the pixel in exposure pattern 4 with 256, thedifference image generating can be performed, whereby the problem ofsaturation can be resolved.

Also, with another method, saturation is detected, and a differenceimage is not generated for the saturated portion. For this saturatedportion, pixel value setting is performed by a dynamic range expandingmethod that is disclosed in PTL 3, for example. With such processing,for example, the distortion reduction effect is weakened, but thedynamic range can be expanded.

5. Description of Advantages of Processing of the Present Disclosure

Next, advantages of processing according to the present disclosure willbe described.

In order to show the advantages of the present disclosure, focal-planedistortion will be described with reference to FIGS. 24A through 24C.FIG. 24A is an image example at the time that a still object is imaged,and FIGS. 24B and C are image examples at the time that an object movingin the horizontal direction from left to right is imaged. FIG. 24B is animage example at the time of imaging with a global shutter (whereinfocal-plane distortion does not occur), and FIG. 24C is an image exampleat the time of imaging with a focal-plane shutter. As shown in FIG. 24C,object distortion occurs with a focal-plane shutter operation.

As a comparison with the related art, a focal-plane distortion reductioneffect with a method shown in Japanese Unexamined Patent ApplicationPublication No. 2006-148496 (or FIG. 12A) will be shown. In the methodin Japanese Unexamined Patent Application Publication No. 2006-148496,the three images shown in FIGS. 25A through 25C are used to reduce thefocal-plane distortion. The difference in readout time by row is takeninto consideration, a blend coefficient is computed, and one image isgenerated from three images.

The distortion reduction processing result from the Japanese UnexaminedPatent Application Publication No. 2006-148496 is shown in FIG. 26. InFIG. 26, the position of the center of gravity of the object isadjusted, but the upper and lower edges of the screen can blur greatly,and the overall reduction effect is not great.

The focal-plane distortion reduction effect according to the presentdisclosure will be shown. As described with reference to FIG. 11, FIG.12B, and FIG. 13, four exposure pattern images are generated first fromone image made up of four exposure times, and four timing imagesequivalent to images shot with four different timings are generated as adifference image of four further exposure pattern images. The distortioncorrection processing unit generates a corrected image wherein thefocal-plane distortion is corrected by a synthesizing processing ofthese timing images.

For example, a specific example of multiple timing images (differenceimages) is an image such as that shown in FIG. 27. The timing images aregenerated as difference images of multiple exposure pattern images,whereby the exposure period for each of the timing images is a shortperiod, and a dark image is obtained, but many images with littleblurring and a short time spacing can be obtained. The corrected imagemade up of the pixel values computed based on the image synthesizingprocessing using the image in FIG. 27, e.g. the pixel value computingprocessing above with reference to FIG. 13 becomes an image such as thatshown in FIG. 28. That is to say, by performing reduction processing offocal-plane distortion, FIG. 28 can be obtained.

As compared to the image shown in FIG. 26 which is the result of amethod of related art, the image shown in FIG. 28 which is the result ofthe present disclosure can be confirmed as having an extremely smallamount of irregularities in distortion reduction by the position on thescreen, and as having a great distortion reduction effect.

As shown above, the image set at exposure times that differ by region isshot with an imaging device, according to the present disclosure, andwith the processing applying the shot image, images that have performedfocal-plane distortion correcting can be generated.

In order to obtain a similar effect with a method according to relatedart, a sensor has to operate at a high speed, but not with the presentdisclosure, so demerits such as increased power consumption thataccompany an increase in operation speed of the imaging device do notoccur. Also, for a similar reason, the storage apparatus serving as aframe buffer does not have to be capable of high speed operations, andpower consumption and apparatus cost can be reduced.

Further, complicated computations such as motion vector calculatingprocessing and the like do not have to be performed, so reduction ofcomputing load and high speed processing are realized. Note that even inthe case wherein the present disclosure and the motion vector areconfigured so as to share the computing processing, the time differencebetween timing images that are to be subjected to motion vectorcomputing is small, and motion vector computing processing can beexecuted with high precision, whereby a highly precise distortionreduction effect can be obtained.

The present disclosure have been described above with reference tospecific embodiments. However, it is obvious that one skilled in the artcan make modifications and substitutions to these examples within thescope and essence of the present disclosure. That is to say, the presentembodiments have been disclosed in exemplary form, and should not beinterpreted in a restricted manner. In order to determine the essence ofthe present disclosure, the Claims should be referenced.

Also, the series of processing described in the Specification can beexecuted with hardware, software, or a combined configuration of both.In the case of executing a processing with software, a program havingrecorded a processing sequence is installed in the memory of a computerthat is built in to dedicated hardware, and processing is executed, or aprogram is installed in a general-use computer that can execute varioustypes of processing, and the processing is executed. For example, theprogram can be recorded beforehand in a recording medium. In addition tobeing installed from a recording medium to a computer, the program canbe received via a network such as LAN (Local Area Network) or theInternet, and installed in a recording medium such as a built-in harddisk.

Note that the various types of processing described in the Specificationare not only executed in a time-series manner according to thedescription, but may be executed in parallel or individually, accordingto the processing capability of the apparatus executing the processing,or as suitable. Also, a system according to the present Specification isa theoretical collective configuration of multiple apparatuses, and isnot restricted to apparatuses with various configurations within thesame housing.

The present disclosure contains subject matter related to that disclosedin Japanese Priority Patent Application JP 2011-038240 filed in theJapan Patent Office on Feb. 24, 2011, the entire contents of which arehereby incorporated by reference.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. A system comprising: an imaging apparatusconfigured to have a plurality of pixels which are arranged in pixelgroups, each of the pixel groups having at least two pixels; and one ormore processors configured to: generate an image based on each pixelsignal from each pixel group of the pixel groups, wherein each pixelgroup of the pixel groups has different exposure times; input the imagewhich has been shot with different exposure times set by region;generate a plurality of exposure pattern images corresponding to thedifferent exposure times based on the input image; generate, from theplurality of exposure pattern images, a plurality of difference images;and generate a corrected image by synthesizing the plurality ofdifference images having different exposure starting times within a setinterval.
 2. The system according to claim 1, wherein each pixel of theplurality of pixels includes a photoelectric conversion element.
 3. Thesystem according to claim 1, wherein each pixel of the plurality ofpixels includes a transistor element.
 4. The system according to claim3, wherein the transistor element is a reset transistor.
 5. The systemaccording to claim 3, wherein the transistor element is an amplifyingtransistor.
 6. The system according to claim 3, wherein the transistorelement is a select transistor.
 7. The system according to claim 1,wherein the input image has been shot with a different exposure startingtime for each row of a plurality of rows and a different exposureduration for each pixel group of the pixel groups.
 8. The systemaccording to claim 1, wherein each difference image of the plurality ofdifference images comprises one or more pixels having pixel values thatare a difference between corresponding pixels of two or more of theplurality of exposure pattern images.